Block 1: Epidemiology

EPI12-1: CRITICAL APPRAISAL

You will first read an article on health surveillance of employees on a South African lead mine* and learn to look at it critically.

The article is:

Health surveillance of employees an a lead mine, 1979-1989. H G van Heerden, JT Mets. South African Medical Journal 1991;79:387-390.

HOW TO EVALUATE THE ARTICLE:

Go to the following points in the article or publication and evaluate each one for adequacy:

Title - appropriate

Abstract/summary - adequate and not misleading

Literature review - adequacy, appropriate and relevant research problem

Motivation

• Problem - what is it?
• Purpose - Why is it being done?
• Are there realistic implementation objectives?

Aim - Concisely stated

Objectives - attainable and precisely defined, do they relate to aim?

Methods:

• Definition of terms and variables adequate;
• Study design clearly stated, appropriate and efficient;
• Study population adequately defined (tree diagram);
• Sampling of study group
  • sample size adequate for power and significance;
  • index subjects;
  • referents;
  • ratio of index/referents;
  • good definition of index and reference subjects;
  • sample representative (covers) the scope of study or target population;
  • selection biases arising from source of index and referent subjects;
  • diagnostic bias;
  • self selection of the ill;
  • healthy worker effect;
  • sick community effect;
  • confounding bias likely and measures taken for this;
  • sample randomly selected or some other method;
  • matching present and adequate;
  • response rate acceptable and attempts to compare responders and nonresponders.
• Measurements
  • blindedness;
  • quality control;
  • precision/reliability/repeatability, any checks;
  • sample size adequate;
  • observers adequately trained;
  • intraobserver difference;
  • validity any checks made;
  • misclassification considered;
  • standardisation of procedures;
  • recall bias considered/likely;
  • calibration of equipment and laboratory quality control;
  • validation against other criteria or sources;
  • sensitivity;
  • specificity;
  • predictive value - positive;
  • negative;
  • interobserver differences analysed, concordance, bias, kappa;
  • data reading errors checked;
  • Biological/subject variation controlled.
• Piloting
  • pretesting of measurement instruments for appropriateness, precision and validity.
• Implementation/data collection
• ongoing quality control for precision and validity, missing or lost data, dropout and death
• Logistics
  • epidemiologic efficiency;
  • cost efficiency;
• Statistical methods
  • appropriate for data;
  • appropriate for study design;
• Resources required
• Nature of the sponsors - interested parties or disinterested parties?
• Ethics
  • informed consent for participation of subjects;
  • feedback to individuals;
  • referral for treatment or follow-up;
  • feedback to interested parties;
  • stakeholder preview for comment;
  • release of scientific information;
  • release of information to general public;
  • other implementation objectives;
  • agreements between researchers;
  • stakeholder contracts;

Results

• sufficient data description;
• population & study group description initially and after exclusions;
• univariate and bivariate statistics;
• computational mistakes;
• arithmetical mistakes;
• adequate treatment of variables;
• causes;
• confounders;
• effect modifiers;
• intermediate variables;
• effects;
• appropriate crude measures to estimate parameters;
• point and interval estimates;
• appropriate crude measures to estimate associations point and interval estimates;
• adjustments made for appropriate summary measures;
• confounding controlled for;
• interaction investigated;
• appropriate statistical testing and presentation of test results in terms of:
  • point estimates of parameters
  • specified interval estimates of parameters
  • test statistic values
  • p-values for test statistics

Discussion

• Awareness of limitations in methods and results especially biases
  • likely causes of negative studies;
  • small sample size;
  • nondifferential misclassification;
  • inappropriate comparison groups;
  • likely causes of false positive studies;
  • differential misclassification;
  • other biases above;
• Reasonableness of interpretation of own and other findings in literature.
• Application of guidelines for assessing causality:
  • strength of association;
  • consistency;
  • specificity;
  • time ordering;
  • biologic gradient or dose-response;
  • biologic plausibility;
  • experimental evidence;
  • analogy;
  • coherence of above evidence;
• Generalizability of findings or External validity.
• Speculative comment identified as such.
• Implementation - appropriate or inappropriate recommendations
  • clarity of implications;
  • efforts made or suggested;
• Publication bias
  • content area for negative or positive findings;
  • nature of publication outlet - overseas, local;

* This work is licensed under a Creative Commons Attribution - Noncommercial Works License.

General Introduction to Occupational Health: Occupational Hygiene, Epidemiology & Biostatistics by Prof Jonny Myers is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 2.5 South Africa License